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Wuxing Cai and Naihuan Jing 

Abstract. We give an iterative method to realize general Jack func- 
tions using vertex operators. We first show some cases of Stanley's 
conjecture on positivity of the Littlewood-Richardson coefficients, and 
then use this method to give a new realization of Jack functions. We 
also show in general that the images of products of Jack vertex opera- 
tors form a basis of symmetric functions. In particular this gives a new 
proof of linear independence for the rectangular and marked rectangular 
Jack vertex operators. Finally a generalized Frobenius formula for Jack 
functions is given and used for an evaluation of Dyson integrals and even 
powers of Vandermonde determinant. 



1. Introduction 

Jack symmetric functions play a fundamental role in the Calegero-Suther- 
land model in statistical mechanics [F] and they also appear as extremal vec- 
tors in the Fock representation of the Virasoro algebra [ MY|, ICJ1| . From 
algebraic and combinatorial viewpoints, explicit realization of Jack func- 
tions is a desirable task and an interesting question. In the special case of 
Schur functions the explicit realization is the Jacobi-Trudi formula, which 
has a long list of applications ranging from Schubert calculus in algebraic 
geometry to various combinatorial problems. The Jacobi-Trudi formula for 
rectangular Jack functions was known |Maj . and it also reveals the connec- 
tion with a special case of Dyson's conjecture for constant terms and even 
powers of Vandermonde determinants [BBLj . 

In this paper we first settle a long-standing problem of realizing Jack 
functions of all shapes using vertex operators. The realization is based upon 
our earlier work of reformulation of Jack functions for rectangular-shapes 
using vertex operators [MY|, IFF|, ICJlj . The key idea behind our approach 
is to realize the general Jack function by a filtration of rectangular shapes 
and then introduce a procedure to encode the filtration in the Fock space. 
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We would like to remark that this new vertex operator realization of classical 
symmetric functions is different from earlier realizations of Schur and Hall- 
Littlewood polynomials [ Jl\ IJ2] . 

As an application of the new iterative vertex operator realization of Jack 
functions, we derive a generalized Frobenius formula expressing a Jack func- 
tion as a linear combination of power sum symmetric functions. This formula 
is new even in the case of zonal spherical functions (in both orthogonal and 
symplectic cases). 

As our second goal, we study the connection between the contraction 
function of the Jack vertex operators for the rectangular shapes and Dyson's 
constant term conjecture, and evaluate more terms in the generalized Van- 
dermonde determinant or even powers of the latter. One of the key steps is 
to compute the terms using our earlier work |CJlj on Jack symmetric func- 
tions of rectangular shapes. Explicitly, vertex operators provide an effective 
tool to compute all coefficients of z^^^ ■ ■ ■ z~^'' z^^_-j^-^ ■ ■ ■ z^^iZg^ + j < s), 
in the product ni<i^j<s(l " where A = (Ai • • • Aj), ^ = (/^i, • • • ,fij) 

are two partitions of equal weight. 

Our third goal is to prove that the products of Jack vertex operators 
form a basis in the Fock space associated to the Jack functions. This is 
deeply rooted to a basic question in the representation theory of higher level 
representations of affine Lie algebras. Lepowsky- Wilson's work on Rogers- 
Ramanujan identities was partly related to linear independence of some 
products of vertex operators similar to our Jack vertex operators. Our case 
is simpler as we can prove a general result that is equivalent to a statement 
for products of vertex operators in arbitrary level representation of the affine 
Lie algebra 5^2 [LB ICJl] . 

Another by-product of our approach is that we can prove several non- 
trivial cases of Stanley's conjecture about the Littlewood-Richardson coeffi- 
cients. We first give a formula on the product of two Jack functions of dual 
shapes, which can be viewed as a Jack case generalization of Schubert-like 
product in the cohomology of the Grassmannian variety. This turns out to 
be a key in computations about Stanley's conjecture. We then generalize 
the result to marked rectangular (rectangle with part of the last row/column 
removed) case. 

This paper is organized as follows. In section 2, we recall some basic 
notions about symmetric functions and collect a few results from |CJlj for 
later use. In section 3 we first give a refinement and a generalization of 
the main result of [CJlj and a new method to compute the coefficients 
of the monomials in the generalized Vandermonde determinant. Then we 
prove that the vertex operator products form a basis in the Fock space of 
symmetric functions, and their basic properties are also listed. Section 4 
treats special cases of Stanley's conjecture, and then solves the problem of 
realizing Jack functions by vertex operators in general. 
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2. The realization of Jack functions of rectangular partitions 

We first recall basic notions about symmetric functions. A partition 
A = (Ai, A2, • • • , A5) is a sequence of nonnegative integers in non-increasing 
order. Sometimes we also write A = 2"*^ . . . )^ where mi = mi{X) is the 
multiplicity of i occurring among the parts of A. The length of A, denoted as 
/(A), is the number of non-zero parts in A, and the weight |A| is irrii. It is 
convenient to denote m(A)! = mi!m2! ■ ■ ■ , z\ = Yl^^i i"^^mi\. A partition A 
of weight n is usually denoted by A h n. The set of all partitions is denoted 
by v. For two partitions A = (Ai, A2, • • • ) Siud fi = (ni, /U2, • • • ) of the same 
weight, we say that A > /x if Ai -|- • • • + Aj > //i -)-••• for alH, this defines a 
partial order (the dominance order) in V. If mi{pL) < mi{X) for all i, we say 
that /X is exponentially contained in A or contained by multiplicity and we use 
the notation ^ c' A. Partition A\;U is then defined by mi{\\ii) = mj(A) — 
m,i{fi), and we also define = Hi (m'(^j)' Sometimes we also identify 

A with its Young or Ferrers diagram ^(A) = {{i,j)\l < i < ^(A), I < j < Aj} 
which is a left justified diagram made of 1{X) rows of boxes (or squares as 
being called by some authors) with Aj boxes in the ith row. A partition of the 
form A = (r*) is called a rectangular partition or a partition of rectangular 
shape. If the Young diagram of is a subset of the Young diagram of A, 
we write /Lt C A, define A — ^ to be the set difference Y{X) — Y{i^). We 
call A — /U a horizontal n-strip if there are n boxes in A — and with no 
two boxes in the same column of A. The conjugate of A, denoted as A', 
is a partition whose diagram is the transpose of the diagram of A, thus A^ 
is the number elements in the set {j\Xj > i}. For a box s = {i,j) € A, 
the lower hook-length /i^(s) is defined to be a~^{Xi — j) + {X'j — i -|- 1) 
in the rational field Q(q;) of indeterminant a, and the upper hook-length 
hl{s) = a-\Xi-j + l) + iX'j-i). 



Example 2.1. Fig.l (a) is partition A = (5,3,3,1), Fig.l (b) is the 
diagram of partition A'. For a = (1, 1), b = (2, 2) G i^(A), we have h^{a) = 
4q:~^ -I- 4, ^^(6) = 2a~^ + 1. As shown in Fig.l (c), the coefficient of 
(resp. constant) is the number of boxes the solid line passes horizontally 
(resp. vertically). 
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Fig.l (a) 



Fig.l (b) 



Fig.l (c) 
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We sometimes write partition A = (a"^a2^ • • • a""), with ma^(A) = rij > 
0, and ai > 02 > • • • > fls > 0. For each z = 1, 2, • • • s, we replace the part Oj 
of A by Cj+i, setting a^+i = 0, therefore obtaining a new partition, denoted 
as X{i). Equivalently, 

X{^) = {a-.^ar ■ ■ ■ al^"/ «m""^^«r;? ' ' ' 

Then we define the ith corner of A as Ci(A) = A — A(i). Cj(A) is of the same 
shape as the rectangle ((aj — aj+i)"*). A has s corners, we write ndX) = s 
as the number of corners of A. nc(A) = 1 if and only if A is a rectangular 
partition and in this case the unique corner is A itself. For a partition 
v = (i/i, • • • jVi) with 1^1 < — Cj+i, I < rii, we define the upside-down v of 
the Cj(A) as the union of Bj for j = 1, 2, • • • , / , where Bj are the rightmost 
Uj boxes on the (rij + 1 — j)th row of Cj(A). For example, we can write 
A' = (4,3,3,1,1) as {A^S'^l'^), A'(2) = (4^1^) and the second corner of A' 
is C2(A') = A' - A'(2) = {(2, 2), (2, 3), (3, 2), (3, 3)}, i.e. the boxes marked 
by stars. An upside-down (2, 1) of the second corner of A is marked by the 
minus sign in Fig.l (a). 

Definition 2.2. For a subset 5 of the Young diagram of A, define hl{S) 
as rises' ^*('^)' similarly for h*^{S). For a pair of partitions C A, we 
say that a square s is based if s is in a column which contains at least one 
square of A — ^, and it is un-based otherwise. We denote /ib(resp. A;,) as the 
set of the based squares of //(resp. A), and Hu (resp. Xu) as the set of the 
un-based squares of /i (resp. A). 

For a set (finite or infinite) of variables zi,Z2,--- and a partition A, 
we define z"^ = z^^ Z2^ ■ ■ ■ . For a Laurent polynomial F{z,w) in vari- 
ables zi,Z2,--- ,wi,W2, - ■ ■ , we denote the coefficient of monomial z^jw^ 
as C^A i^^i.F{z, w). 

The ring A of symmetric functions in variable Xj's is a Z-module with 
basis m\ = (m(A)!)~^ Yl ' ' ' ' where the multi-indices (ii, • • • , ik) run 
through (Z>o)'^. We define p„ = m(„) = X^j>;^ xf. The power sum symmet- 
ric functions px = ■ ■ -px^, X £ V, form a basis in Aq = A (g)^ Q. 

Let F = Q(a), the field of rational functions in indeterminate a. For 
two partitions A, /x G "P the Jack scalar product on A^r = A (8)z F is given by 

(1) {px,Pi.) =Sx,^,a-^^^^zx, 

where 5 is the Kronecker symbol. Here for convenience we use instead 
of a in light of Sections 2 and 3. 

The Jack symmetric functions or Jack polynomials |Ja] Px{a^^) = 
Px{xi,X2, • • • ; a~^) for A G P are defined by the following conditions: 

Pxia'^) = CA^(a^^)m^, 
{Px{a-'),Pf,{a-')) = OfoT X^fi, 



JACK VERTEX OPERATORS AND REALIZATION OF JACK FUNCTIONS 5 

where cx^{a^^) € F (A,/x G "P) and cx\{a^^) = 1. Note that symmet- 
ric functions mx,px, Px{a~^), and the following (5a(«~^)) Jxict^^) are all in 
variable xi,X2, - ■ ■ ■ The existence of Jack symmetric functions was proved 
in |M] . Further properties of Jack functions are studied in [S], |KS| and 
[H] . In this work we will also need two other normalizations. One is 
the dual Jack functions Qx{a~^) = {P\{a'^),P\{a^^))^^Px{ct'^)- An- 
other is the integral Jack function Jx{a~^), which is defined by Jx{a~^) = 
\X\l{c^ Px{a~^). Let us recall Fieri formula of Jack functions for later 

use. 

Theorem 2.3. [SJ For partitions fx, A and non-negative integer n, the 
scalar product {J^{a~^)Jn{a~^), Jx{a^^)) ^ if and only if X — fjL is a 
horizontal n-strip. And in this case 

{J^{a~^)Jn{a-'),Jx{a~^)) = nla-'^hl{Xu)ht{Xb)h;{i2t)h^{fiu)- 

For later use, we would like to introduce some terminology here. For an 
operator S on symmetric functions, if {J^,S.Jx) is non-zero, we say that S 
can remove X — /j, from A; otherwise we say that S can not remove A — /x 
from A. For example, J„(a~^)* can remove a horizontal n-strip from A if 
Ai > n. Furthermore if A is of rectangular shape, then Jn{a~^)* can only 
remove the n boxes on the right side of last row of A. 

The generalized complete symmetric functions of A^r are defined by 

(2) qx{a-^) = QxAo^'')Qx,{a-^) ■ ■ ■ QxA^'')^ 
where Qn{a~^) = Q(„)(a~-'^) has a generating function given by 

oo „ 

(3) e:cp(^^ap„) =Y,Qn{a-^)z^. 

n=l neZ 

Thus (5o(a^^) = 1 and Qn{a'^) = forn < 0. We list some of the useful 
properties of and Jx as follows. 

Lemma 2.4. [SJ For any partition X, v, one has 

{qx{(y'^),my) = 5x,v, 

Qx{a'^) = qx{or^) + XI (^xi,{or^)q^i{a'^), 

Jx{a-^) = /i^(A)Pa 

JA(a-i) = h\{X)Qx{a-^), 

where ax^t{a~^) e F = Q{a). 

2.1. Fock space V and rectangular Jack functions. Let us recall 
the realization of rectangular Jack functions on the Fock space of the rank 
one vertex operator algebra. Let 

y = Sym(/i_i,/i_2,---)®C[^Z], 
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where Vq = Sym(h^i, /i_2i ■ ■ ■ ) is the complex symmetric algebra generated 
by the variable /i_„(n S N) and C[^Z] is the group algebra of with 
generators {e^'^jn G |Z} with multiplication given by e"^'^e'^^ = e(™'+")'*. 
The underlying Heisenberg algebra is the infinite dimensional Lie algebra 
generated by /in, n G Z — {0} subject to relations 

(4) [h^,hn\ = ma'^6jn-nl- 

For convenience, we set /iq = 1 € Vq- The associated scalar product of V is 
given by 



where for partition A = (Ai, A2, • • • )) ^-A is defined to be h_\^h_x2 ' ' ' • 
For an operator SonV (or Ai;'), the conjugate of 5, S* is defined by 

{S.u,v) = {u,S*.v), u,v gV. For a symmetric function /, we also denote 

the multiplication operator 5 — >• by the same symbol /. It can be shown 

that /i* = h-n for n G Z. 

Note that B = {h_x 'S> e"'^\X G "P, n G ^Z} is a basis of V, we can define 

a linear map T from y to Ai? by 

r(/i_A ® e"'^) = px. 

Comparing formulae ([TJ and ([5]), we know that T preserve the scalar prod- 
uct, i.e. 

{T{u),T{v)) = {u,v), 

for every pair u,v G V. 

We then define the vertex operators 



(6) X{z) = exp ( — g/i-n j Aea:^ ( 2a/i„j = X„ 

n=l n=l nSZ 

°° n 

(7) y(z) = exp( ^ —ah.n) = 



00 



(8) Y*{z) = exp^Y.-'^H = E^n^""' 

n=l 

where 

/in-v <8) e'^'^ = (a'-^n-Q^v) (g) e'*'' if n > 0, 
hn-v ® e"^ = {hnv) ® e""^ if n < 0, 

and 

(9) A.t; O e"'' = z("+3)2«t> O e("+^)''. 

Remark 2.5. Comparing ([3]) with ([7]), we know that r(y_„) = (5„(a~-^) 
for n G Z. Moreover, because = we know that Y* is actually the 
conjugate of for n G Z. 
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The following result provides a realization of Jack functions of rect- 
angular shapes. For a partition A = (Ai,--- ,A/) of length I, we denote 
X_A = X-Ai^-A. • • • X^x„ and y_A = Y^x.Y^x, ■ ■ ■ Y^x.. 

Theorem 2.6. [CJT] For partition X = {{k + 1)" , {kY) with k G Z>o, 
s S Z>o, t G {0, 1} and a G Z>o, we have 

where Cs^i(a) is a function of a, and Cs^i(l) = (— l)**^*~^^^*^/^s!. 

We also need the following proposition for later use. 

Proposition 2.7. |CJ1| For positive integer a and partitions X, fj, of 
the same weight, we have 

(10) (X_A.e-'(^)^/^r-;..e'W'^/2^ = {-l)<'-'^"/^C,./^,.H^{Zs,Wt), 

«;/ierei/,(Z,,Vri) = n.>.^,>i(l-^,^r'rn;=in-=i(l-^.^«r')~"- More- 
over, is zero if //i > Ai . 

3. A basis for symmetric functions 

3.1. Further results on Jack vertex operator. Suppose a G N. A 
special case of Dyson's constant term conjecture, proved by Wilson |Wi] 
and Gunson |Gu| independently, says that the constant term of 

n (1-^)" 

is {sa)\{a\)~^ , thus we have the following refinement to Theorem 12. 6[ 
Corollary 3.1. In Theorem [1751 we have 

cs{a) = c,,o(«) = {-ir^'-'^l\sa)\{o^r' 
in the case of t = 0. 

Remark 3.2. If s > 2, then Cs{a) is not a rational function of a. 
Proof. Recall that we have [CJl] 

(11) T(A_AiA_A.---A_A..e"'^) 

where Di is the lowering operator defined as Di.Qx^{a~^) ■ ■ ■ Qx^{a~^) = 
Q\-^^{a~^) ■ ■ ■ Q\^-l{a~^) ■ ■ ■ Qx^{a~^). Note that for rectangular A, only 
constant term contributes to qx, the statement then follows from Dyson's 
constant term conjecture. □ 
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From the proof we also have the following equation (I12p which links even 
power of Vandernomde determinant to rectangular Jack functions: 

(12) A^g(,.)(a-i) = |^g(fc.)(a-i). 



In the context of Theorem 12.6^ there is also a combinatorial formula in 
terms of p^'s for the products of vertex operators (Theorem 3.2 in [ CJlj ). 
Combining this with Corollary 13. H we get a new expression for rectangular 
Jack functions. For clarity and latter use, let us first give the following: 

Definition 3.3. For rectangular partition R = {k'^), positive integer a 
and ks, define 

(13) 

- ( irMs-iy2 (air y^-pr (-2a)^(-V m(M-i) 

where the sum is over sequences of partitions fj, = (fi^, fi'^, ■ ■ ■ , /i*) and = 
(z^^, zv^, . . . , z^*) such that fjP = (0), ^u"* = /x, z^* c' /x*, |^*| = i{k + (s — i)a), 
and ^*\z^* C' 

Corollary 3.4. For positive integers k, s, a, we have 
(14) Q(fc.)(Q"^) = g(k'^),^,{a~^)Pfi- 

Note that we can expand Jack functions in terms of pa's by other ap- 
proaches, thus Corollary 13.41 also gives various different combinatorial iden- 
tities correspondingly. As a simple example, we set (k^) = (1^), and get the 
following nontrivial identity by considering coefficients of on two sides of 
identity (fTil). 



Corollary 3.5. For positive integer a one has 
^ {-2a)^-' (-2a)'(^) /^m(l/)^^ _ (2a)! 4a2 



(2-i)! ^ Zy \m(V)J (a!)2a-hl' ' 

Since we realized the Jack functions of rectangular shapes in t = case of 
Theorem l2.6t we can further generalize this result to the case of a rectangular 
tableau plus one row. 

Corollary 3.6. For partition X = {{k + 1)") with k e Z>o, s G Z>o, 
and integer n with {) < n < k + 1, we have 

T(y!„X_A.e'^'^/2) = 4,fc(a)g((fc+i).-i,fc+i_„)(a-i) 

where ds^k{ot) 7^ is a function of a. 
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Proof. By Remark 12.51 it is immediate that 

where we used Theorem 12.61 and its notation. By Fieri formula Theo- 
remlZa {Q.^ia-^)* .Qxia-'),Q^,ia-^)) = {Qx{a-^),Q-nia-^)Q^.{a~^)) 
is non-zero if only if A — /i is a horizontal n-strip. But A = {{k + ly) 
is of rectangular shape, there is a unique // satisfies this property, i.e. 



3.2. The even power of Vandermonde determinant. The lower- 
ing operator A" = ni<i5^j<s(l ~ i^ essentially even power of Vander- 
nomde determinant in the sense that 

Yl {Di - Djf'^ = {Di--- Z),)"(^-i) (-l)^(^-i)"/2 A^. 

l<j<i<s 

Thus the following corollary can be used to study the generalized Vander- 
monde determinant. Recall that by Lemma 12.41 the transition matrixes be- 
tween q\{a~^ys and Qa(«~^)'s are triangular matrixes, if we arrange them 
by the partitions with a total order which is compatible with the dominance 
order. 

Corollary 3.7. Set Q(^k^){a''^) = Yy^i>{k'') "'i^i'^~^)9t^i'^~^) ■ ^^r /3 = 
{bi, ■ ■ ■ ,bs) (z Ij'^ such that hi + ■ ■ ■ + bs = and bi > b2 • • • > bg > —k, we 
have 

where iifi = {k^) + j3 is a partition. 

Proof. Consider coefficient of g^g(a~^) on two sides of p^ . and no- 
tice that including ■ ■ ■ D^" ^ there are s\/m{^is)\ terms contributing to 
qfj,p{a~^) when A" acts on q{ka^{a~'^). □ 

Remark 3.8. Dyson's constant term corresponds to the special case that 
/? = G Z''. Note that we have an iterative formula |CJ2|, ILLMj for the 
coefficients a^(a~^)'s. Hence these coefficients can be evaluated effectively 
using vertex operator calculus. The following gives some examples on this. 

Proposition 3.9. For positive integers i, s, a with 2i < s, the co- 
efficient of D^^ ■ ■ ■ ■ ■ ■ Dg in Laurent polynomial A" = 

(16) C^-...,,-z....,.D...,....^.-A? = |^^!(-a)^ri(l + {s-j)a)-\ 
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And we also have 

(sal 2a^-{l + is-l)a){l + {s-2)a)a 
{sa)\ 2(2 + (s - l)a)a^ 



a 



^Di^D^ ^Dl-^s (1 + (s - l)a)(l + (s - 2)a)(3 + (2s - 3)a) ' 

Remark 3.10. In [K], Kadell evaluated the coefficients of ziz^^, xxziz^^-^z'^'^ 
and 21^2-^7^ in Dyson's Laurent polynomial ni<i5>^j<s(l ~ Here 
we consider the special case when the exponents are identical, but we 
can compute the coefficient of z^^^ ■ ■ ■ z~^^ z^^'_-j^^ ■ ■ ■ z'^'^^Zg^ for any pair of 
partitions A = (Ai • • • Aj), fi = (/ii, • • • with |A| = |/i| and i + j < s. 

Moreover the steps of computation depend on A and fi but not a and s. 

3.3. The independence of the Vertex operator products. For a 

partition A, denote T{X^x-e~^^^'^^^'^) as X^xia'^), and set 

X'_x = X'_x{c^-') = q(A)(a)-iX_A(«-'), 

where Cs{a) = ||^(— l)'^*-'^""'^^"''^ is defined in Corollary 13. 1[ Note that in 
general Cs(a) is not a rational function of a, and that is why we factor it 
out from X_x{a~^). We will prove these ^^'s form a basis for the space of 
symmetric functions over Q(a). First, Corollarv 13.11 implies that X'_^ is in 
Ap for rectangular partition A. In fact, this is true for general partitions. 

Lemma 3.11. For any partition X, X'_^ is a symmetric function over 
Q(a), i.e. X'_^{a-^) G A^. 

Proof. In (fTTI) . the coefficient of the term ^/^(a^^) for each fi is Cs{a) 
multiplied by some rational function (which depends on ^) of a by Corollary 
13.71 thus we can factor Cs{a) out and rewrite T{X- the desired 

form. □ 

Second, we can prove that the X_;^(l)'s are linearly independent with the 
following detailed analysis about the specification of a = 1 in Proposition 

Lemma 3.12. For 

s t 

l<iy^j<S 1 = 1 j = l 

let A = (Ai, • • • , As) be a partition of length s, fi = (^i, • • • , fit) be a partition 
of length < t. Then C^A/^^-f^ 7^ implies A > /x. Moreover C^xj^x.H is a 
positive integer. 

Proof. We use induction on s. The case of s = 1 or 2 is trivial by 
Proposition 12.71 Assume that it is true in the case of s — 1. Then in the 
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case of s, we have H = ^l^i Hi^i, where 

H^,l =(1 - ziw-')-^ . - ziz-') . - ziwjY' 

k^l j=2 

■ Hi{zi,--- ,zi,--- ,Zs;w2,--- ,wt), 
where the hat ^ means omission. By induction assumption, we know that 
any term z'^^ ■ ■ ■ zf ■ ■ ■ z^" / (wg^ ■ • • Wj' ) from 

^^i,z,i = Hi{zi,--- ,zi,--- ,Zs;w2,--- ,wt) 

has the property that aj^ -\ \-aj^ > 62 H 1- h+i for i distinct numbers 

ji, • • • , ji from the set {1, ■ ■ ■ , «}■ The term z^^ ■ ■ ■ z^' /{wf^ ■■■wf) 

from Hi^i/Hi^i^i has the property that Q + Cj^ + ■ ■ ■ + Cj. > di + ■ ■ ■ + di+i- 
Multiplying two terms together, the term 

z'^^^^' • • • • • • z,"=+"V(<^wi^^+^^ • • • 1/;^^'+^') 

from Hi^i has the property that %i + • • • + Ojj + Q + + • • • + Cj. > 
^2 + • • • + ^j+i + di + ■ ■ ■ + (ij+i. Thus if it is written as z^/w'^ for two 
partitions A, /x , one should have Ai + ■ ■ ■ + A^+i > fj,i + ■ ■ ■ + Hi+i- This 
finishes the proof of the first statement. 

For the second statement, let Ai = ■ ■ ■ = Aq > Xa+i, then the term 
z^/w^ can only come from Hi^i with I = 1, - ■ ■ ,a. The only way it appears 
in Hi i is when we have z^' /w^^ from (1 — ziw^^)"^, with coefficient 1, 

and then z^^ ■ ■ ■ zj^' ■ ■ ■ Zg" /{w2'^ ■ ■ ■ w^") from -ffi,;,i, with coefficient being 
a positive integer by induction, we then multiply them together to get the 
result. □ 

Corollary 3.13. For positive integer n, {X_;^(1)|A h n} is linearly 
independent. 

Proof. By the previous lemma, when expended into combinations of 
monomials, X_x{^) contains only terms with < A, and it contains 

mx- n 

Now we can prove the main theorem of this section. 

Theorem 3.14. {X'_^{a~'^)\X eV} is a basis for Ap. 

Proof. We only need to show that for each positive integer n, the set 
{X'_^{a^^)\X h n} is linearly independent. Suppose it is linearly depen- 
dent, there are relatively prime polynomials fx{a) € Q[a],A h n such that 
EAhn hic^)XU{cx-^) = 0. Setting a = 1 it follows that Eahh /a(1)^-a(1) = 
0, or X;Ahn/A(l)fcA(l)~^^-A(l) = 0. Note that the coefficients /a(1) are not 
all zero since a—1 is not the common devisor of the /A(a)'s. This contradicts 
to the fact that the X_a(1)'s are independent. □ 

For convenience we list the properties of this basis (generated by vertex 
operators) in the following. 
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Proposition 3.15. The basis X'_^{a ^) 's has the following properties. 



(c) X'_^{a ^) is a linear combination of the q^{a ^Ys with l{fi) < 1{X), 

(d) X'_^{a^^) is orthogonal to q^{a~^) if l{^') > /(A'). 

Proof. Equality (a) comes from Corollary 13. 1[ Equality (b) comes from 
t = 1 case of Theorem 12.61 with the coefficient fixed by using i = 1 case of 
formula (jl6p in Proposition 13.91 Property (c) is from formula (jlip in the 
proof of Corollary 12.61 Property (d) comes from Proposition 12.71 □ 

Remark 3.16. The vertex operator Xx and X'^ differ by the Dyson's 
constant term in ni<j^j<«(A)(-'- ~ XiX~^)°' up to a sign. 

To end this section, we give a new vertex operator realization of Schur 
functions of rectangular shapes with part of the last column removed (cf. 
|J1] ). Note that < Z(A) is a necessary condition for to appear in the 
expansion of X^x{a^^). As a byproduct of Lemma 13.121 we have 

Corollary 3.17. For partition A = ((/c+l)"*, A;*) of length s+t, ^-a(1) = 
c\Sx for some non-zero constant c\. 

4. Stanley's positivity conjecture and Filtration of rectangles 

From now on, the parameter for Jack functions is assumed to be a, sub- 
sequently the scalar product is the original one, i.e. {p\,Pfj.) = 5x^z\q!''^^\ 
The following conjecture about Littlewood-Richardson coefficients of the 
Jack function was made by Stanley |S], referred to as Stanley's LR conjec- 
ture. Denote Z>o[a] as the set of polynomials in a with non-negative integer 
coefficients. 

Conjecture 4.1. The LR-cofficient C^^{a) := {J^Ju,Jx) is in Z>o[a] 
for any triple of partitions A). 

Remark 4.2. Stanley's Fieri formula for Jack function implies his con- 
jecture in the case of = 1. One the other hand, using [KSj, we know that 
C^^(a) G 1j[a]. If the classical Littlewood-Richardson coefficient C^^ = 1, 
Stanley further conjectured a form for C^j^{a) which he verified under a 
technical condition, and this contains part of the following Theorem 14.71 
Furthermore Stanley showed the latter conjecture is true for z/ = (2, 1) un- 
der the condition C^,^ = 1. We will show that Conjecture 14.11 is true for 



4.1. Rectangular and marked rectangular cases. In the following 
we prove two special cases of Conjecture 14.11 

Definition 4.3. For a rectangular partition A = (r^), and a partition 
1/ = (i/i, 1/2, • • • , I's) C A, define the complement partiton of in A as 17 = 
X-' = (^r - Vs,r - Us-i, ■■■ ,r - i^i). 




1/ = 



(2,1). 



JACK VERTEX OPERATORS AND REALIZATION OF JACK FUNCTIONS 13 



Example 4.4. For A = (6^), v = (6,5,3,3,0), we have \ -' v = 
(6,3,3, 1,0). As shown in the following Fig.2 (a), the boxes below the bold 
line forms an upside-down v of A. The partition above the bold line is X—'u. 
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Fig.2 (a) 



Fig.2 (b) 



Fig.2 (c) 



Note that u C (r*) is equivalent to that l{u) < s and ui < r, and in 
this case, the Young diagram of (r*) — ' u is the set difference of Y{X) with 
the upside-down v of A. We have the following simple properties for the 
complement of partitions with the proof being obvious thus omitted. 



Lemma 4.5. For a rectangular partition A 
A, we have 



{r^), and partitions v^uj C 



V = V, 
CO D v 



OJ > u, 
uj CV. 



Lemma 4.6. For v C X = (r*), {J^Qv, J\) 7^ implies lo > u. 

Qti-Jx^ the action of q* • ■ ■ q^, on Jx has the 



+ Vi squares in the last i row(s) of A. Here we mean 
i^i-^A = Z^o) CujJu-, if / 0, then u is coming from A 



Proof. ql.Jx = 

effect of remove vi + 
that setting q*_^ ■ 

by removing vi + \-Vi squares in the last i row(s) of A. Thus in the first 

s — i row(s), there are at most Vi+i + ■ ■ ■ + Vs squares that has been removed. 

Thus this kinds of u satisfies wi -|- • • • -|- cos-i > r{s — i) — {vi+i -\ +Vs) = 

{r — Vs) + • • • + {r — Vi+i), which means oj >V. □ 

The following proposition claims the correctness of Stanley's L-R con- 
jecture in the case that A is of rectangular shape. 

Theorem 4.7. For rectangular partition X = (r*), {Jfj,Ju, Jx) 7^ and 
only if u d X and = V. Furthermore, in this case we have {Jij,Ju,Jx) = 
h*{iJ,)h*{u)hi{iJ,)h2{i'), where 

(«.i)G/i 

^2M= n [{s-i + l) + a{j-l)]. 

Remark 4.8. Note that except the familiar terms ht{ii) and hKv), 
there are two new terms /i2^(/x), (^) C^^{a). For instance, let us see 
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A = {6^),iy = (6,5,3,3) as in Example 14.41 The partition jjL = \ —' v lies 
above the bold line in Fig. 2 (a). Fig. 2 (b) shows the familiar terms, the box 
a in II contributes 2 + a, and box b in (6^) — /i contributes 3 + 4a. Fig.2 
(c) shows the new terms, where the box a contributes 1 + 5a and the box b 
contributes 5 + 2a. So, in (b) or (c), we associate a linear polynomial of a to 
each box, the number of the boxes where the solid line passes horizontally 
(resp. vertically) is the coefficient of a ( resp. the constant). 



Proof. Set = X^(^>i, Co^go;, (J^iJuiJx) implies at least one of 
[Jfiflu)-, J\) 7^ for some UJ > v, thus /i > cJ > 17 by Lemma WM and Lemma 
14.51 Similarly, we have v >'p-, which leads to 17 > by Lemma |4.5[ Now 
/i = 17 is set up. 

Now we can set J*.J\ = Cx^yJ^, and then {J^Ju,Jx) = {Ju,Jt-Jx) = 
Cx^u{Ju, Ju) = C\^jjh^{j')h^ijy). We need to determine the coefficient C\^y. 
We can first write J* = Ylioj>v^i^ltj with Cy = /i*(z^) by Lemma [2.41 then 
the only term that counts is c^q* by Lemma 14.61 The action of q*^ on J\ 
removes vi squares in the last row of A, resulted in, say Cx^y^^Jj^, where 
Cx,ui is explicitly known by Stanley's Fieri formula for Jack function. Then 
we apply the action q*^ on Jjj;^- There maybe more than one ways to re- 
move 1^2 squares horizontally from (i^i), but the only way that contributes 
to J-jj is to remove 1^2 squares in the second last row, which results in, 
say Cx^[yT^^U2) J[ux U2) ^ with Cx^(ui,iy2)j which is known by Stanley's formula. 
Continuing this process, we find that Cx,u = hKv) ■ (Cx^uiCx^^uj^^u^) " ') — 
hl{u)-{h\{iJi)hl{v)/hi{V)). ' ' ' ' □ 



Remark 4.9. 1. It is interesting that the expression is symmetric in // 
and I/, but the symmetry is not at all clear from this form of the formula. 
2. By the same method, we can easily generalize this theorem to Macdonald 
case using the Fieri formula. 



Corollary 4.10. For a marked rectangular partition A = {r^~^,r — n), 
the matrix coefficient {J^Ju, Jx) ¥^ if and only if {{r^) —' v) — ^jl is a 
horizontal n-strip. In this case {J^Ju, J\) is a product of nonzero linear 
polynomials in Z>o[a], Explicitly, denoting V = {r^) —' v, we have 



(17) 



n 



1' 



r(r - 1) • • • (n + 1) nr=o + 



K{liu)h*{lib) 



where fib and ( resp. fiu o,nd Uu) are the sets of based (resp. un-based) 
boxes of fi and V respectively for the pair fi CV. 
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Proof. First we have 




By Theorem 14. 7| it does not vanish if and only if ((r*) — ' z^) — /i is a horizon- 



formula of Jack functions [Sj. Combining this with theorem 14.71 gives the 



Remark 4.11. This proposition implies that Conjecture 14.11 is true if 
A is a rectangle with part of the last row removed. We can also use this 
method to treat the case of rectangular partitions with part of the last 
column removed, and prove that Conjecture 14.11 is true if A = {{k + l)*/c*) 
with k,s,t E Z>o. 

We can generalize Theorem 14.71 to the following situation, which is also 
useful when we consider the ly = (2, 1) case of Stanley's conjecture. 

Proposition 4.12. Assume that {J*.J\, J^) / 0, and A — /u is an upside- 
down uj of a corner of A with \oj\ = \v\, then uj = v and {Jx, Ji_lJu) is the 
product of linear polynomials of a with non-negative integer coefficients. 

Proof. As the proof of Theorem 14. 7| we have lo > v. Note that 
(Jx^Ji^iJu) = {Ju-Jx^Jfi) 7^ is equivalent to {Jx' , Jfi' Ju') 7^ 0, and that 
A' — /i' is an upside-down uj' of a corner of A' with |a;'| = thus we also 
have w' > i^' . Subsequently oj = v. The proof for the other statement is the 
same as that of Theorem 14.71 □ 

4.2. V = (2, 1) case. The following proposition confirms the Stanley's 
conjecture for the case of = (n, 1) subject to a simpler technical condition, 
but without the condition C^^ = 1. 

Proposition 4.13. For v = (n,l), {J^Jv,Jx) is a product of linear 
polynomials of a with nonnegative integer coefficients if the boxes of X — fj, 
lie on two rows of X. In particular, for u = (2, 1), {J^Ju,Jx) G ^>o[o]- 

Proof. We first prove the (n, l)-case. Assume that among the n + 1 
boxes of A — ;U, there are n — a boxes in the same row of one corner, and a + 1 
boxes in the same row of a lower corner. We mark the two right-most boxes 
on the two rows by A and B. We also denote t (resp. s) as the number 
of rows (resp. columns) between the rows (resp. columns) with removed 
boxes. See the following Fig. 3 (a). 



tal n-strip. Then the coefficient of J^r in JnJ^ is 




n!a" by Fieri 



corollary. 



□ 



A 




d 



--n — a 



t 



b 
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We can assume that n — o, o + l>l. In fact, if one of them is zero, then 
\ — hes on one row of A, and \ — ^ should be an upside-down (n + 1) of a 
corner of A. This is a special case of the Proposition 14.12] Note that we have 
J{n,i) = (n-l)!a"[(l + nQ!)g„gi-(n + l)gn+i], and {JyJy,Jx) = {Jf„J*.J\). 
Using Fieri formula, we compute the coefficient c of in g*_,_^.J;^. For 
q^ql-Jx, we compute coefficient ai of Ja-a in q\.J\, coefficient 02 of in 
q'^-Jx-A- We also compute coefficient bi of Jx-b in Qi-Jx, coefficient 62 of 

in q^.Jx-B- Then {J^J{n,i), J\) = (n - l)!a"[(l + na)(ai02 + 6162 - 
(n + l)c)](J^, J^) is found to be a product of linear polynomials of a with 
nonnegative integer coefficients. 

Now we prove the case of (2,1). The idea is same as the (n, l)-case. 
Among the three boxes of A — /i, if any two of them are in the same column 
of A, then by Fieri formula only 9i contributes to in J*^ lyJx = a^[(l + 
2a)q2ql — 3q^].Jx, and it can easily be seen that the proposition is true. If 
any two of them are in a same column of A, it is also true by the (n, l)-case. 
Thus we only need to prove the case when the three boxes of A — // are in 
three different corners of A, see Fig. 3 (b). We mark the three boxes of A — 
by ^1,^2)^3) d (resp. c) is the number of rows (resp. columns) between 
boxes Ai and A2. Similarly we define a,b as shown in Fig. 3 (b). A row 
(resp. column) of /x is called a critical row (resp. column) if it lies on a row 
(resp. column) of A which intersects A — /x. Now 

{Jf.J{2,i),Jx) = a^KiR -Rn C)hl{C -Rn C)h*^{fi - R)h>:{fi - C)f{a), 

where R (resp. C) is the union of the critical rows (resp. columns) of /i , and 
/('^) = Yli=Q CjO*, with Cj G Z>o[o, ^) c, d], thus the positivity follows. □ 

4.3. Filtration of rectangular partitions. Using Theorem 14.71 we 
can derive Jack function of any shape starting from Jack functions of rect- 
angular shapes, which were already realized in [CJl] using vertex operator 
technique. 

Definition 4.14. Define maps 9^,(r on P by 9^(A) = (/(A')'^^)), C(A) = 
1K(A)-'A. Define Ri{\) = lH(e:*-i(A)) (i > 1, C°(A) = A). Set s = nc{\) the 
number of corner of A, Rx = {Ri, • • • , Rs) is called the rectangular filtration 
of A. 

Example 4.15. For A = (6,3,3,2,1), note that /(A') = Ai, so i?i(A) = 

1R(A) = (6^). e:i(A) = e:(A) = (6^) -'(6,3,3,2,1) = (5,4,3,3), i?2(A) = 

9^(e:i(A)) = (5^). Similarly, it can be found that ^^(A) = e:(e:(A)) = (2, 2, 1). 
i?3(A) = (2^), C3(A) = (1) and i?4(A) = (1^). Thus the filtration of A is 
R\ = ((6^)) (5^)) (2^)) (l"*^))- The following figure illustrates the pictorial 
construction of the rectangular filtration. The A is the partition above the 
bold line in picture (1), the whole rectangle in picture (1) (with diagonal 
aooi) is Ri. -Ri — A is the subset under the bold line in picture (1), it 
is an upside-down C^(A) = (5,4,3,3) of (the corner of) Ri. The whole 
rectangle (2) (with diagonal 0102) is R2, it is also the sub-rectangle with 
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diagonal 0102 in picture (1). Similarly, the whole rectangle in picture (i) 
with diagonal ai-iUi is Ri. In fact, all rectangle Ri is marked in picture (1) 
with diagonal aj_iaj. 
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(1) (2) (3) (4) 

Fig. 3 rectangular filtration 
Lemma 4.16. For a nonzero partition X, 9^(A) is the minimum rectangu- 
lar partition containing A with respect to the partial order C, and nc(C(A)) = 
nc{X) - 1. 

Proof. If a rectangular partition u = (a^) contains A, then we should 
have a > Ai and b > /(A) which means 9\{X) C i^. This proves the minimum 
of 9^(A). To prove the statement about the number of corners. We write 



A = (a- 



), with ai > 02 > ■ ■ ■ > Os > and ma- (A) = rij > 0. By 



definition of the compliment partiton we have 



(18) 



<t{X) 



ni- 



"1 „n,2 
2 



= ((ai - a,)"^(ai - a,_i)"»-i • • • (ai - 02] 
Thus nc(A^) = s — 1 and the lemma follows because nc(A) 



□ 



We need the following lemma in the proof the realization of Jack func- 
tions of general shapes. 

Lemma 4.17. If Rx = {Ri,--- ,Rs) is the rectangular filtration of X, 
s>2, then {R2,R3, • • • , Rs) is a rectangular filtration of v = £(A). 



Proof. By Lemma 14.161 rac(z/) = s — 1. Thus we can assume that 
Ri, = {Ri,--- ,Rg_i) is the rectangular filtration of i'. By definition, we 
have = = m{e-\(t{X))) = 9\{e{X)) = Ri+i. □ 

We would like to give a direct description of the rectangular filtration 
of a partition. For it = (ni, • • • ,ns) G Z*, define it* = {n\, ■ ■ ■ ,n*) € li^ 
by setting n*.+i = Ei<j<.-i+i = T.i<j<s-i+i^j- For a partition 

A = (a"^a2^ • • • a"^), where ma.(A) = > 0, we call the vector it = 
(ni, 712, • • • ) fis) the multiplicity type of A. 

Prom the Young diagram of A we can easily prove the following lemma. 



Lemma 4.18. For a partition X 
the multiplicity type of X' is~^ = {pi,P2, 
with Os+i = 0. 



(«1 «2 



with ruai (A) = rij > 0, 



,Ps) where pi = Qg+i-i - 0^+2- 
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Lemma 4.19. For it = (ni, ■ ■ ■ ,ns)GZ^(s>2),setr^ = {ns,ns-i ■ ■ ■ ,722). 
Then we have rrt* = (71-2, 7X3, • • • , n*). 

Proof. Set rrt = {mi, 1712 • • • ,ms-i), then mj = n^+i-j and 

i<j<(s— l)-i+l i<j<s-i i+l<fe<s+l— j 

Similarly it can be computed that ?7i2i-i ~ ^2i- 

□ 

Proposition 4.20. For a partition X, assume that the multiplicity type 
A and X' are it = {ni, ■ ■ ■ ,ns), ~^ = {pi,P2, ■ ■ ■ ,Ps), respectively. Then the 
filtration Rx = {Ri, R2r " jRs) is given by Ri = (p*"' )• 

Proof. Set A = (a^^Og^ • • • a"^), with ma^X) = Ui > 0. We make 
induction on the number of the corners ric(A) = s. It is obviously true if 
?2c(A) = 1. If it is true when nc{X) = s — 1, let us prove the case of 7^c(A) = s. 

Note that pi = pi -\ \- Ps = J2i<i<s(o-s+i-i - = ai (os+i = 0), 

and nl = ni-\ h = /(A), we have Ri = {pl"'*^)- 

We set I' = (t{X) then v = {{ai - a^)"=(ai - a^_i)"='^ • • • (oi - 02)'^^). 
The multiplicity type of v and u' are r?t = {ns,ng_i, ■ ■ ■ ,722) and if = 
{ps,ps-i, • • • ,P2), respectively. By the inductive assumption and Lemma 
Ea we have R, = ((^r^), • • • , fe^r*-)) = ((pf^), • • • , (pT'))- Apply- 
ing Lemmas 14. 171 we have Ri = (p*"' ) for i > 2. □ 

Recall that Jack functions of rectangular shapes were realized by ver- 
tex operators in (CJlj . thus the following theorem gives a vertex operator 
realization of Jack functions of general shapes. 

Theorem 4.21. For a partition X, let Rx = (i?i,i?2,-- - , Rs) be the 
rectangular filtration of X and denote the rectangular vertex operator repre- 
sentation of Ri by fi = Qr-, then there is a c'^{a) G Q(a)\{0} such that 

(19) Qx = c'x{a){- ■ ■ {{f:.fs-ir.fs-2r. ■ ■ ■ .f2r.f1. 

Proof. For polynomials Fi, F2 in A^?, we write Fi = F2 if Fi = CF2 for 
some c € F — {0}. We use induction on s = nc{X). If s = 1, it is trivial. 
Assuming (119p holds for the case of s — 1 (s > 2), then 

Q£(A) = (---((/;./s-l)*./s-2)*.-- - ./3)*./2. 

This is due to the fact that {R2, • • • ,Rs) is the rectangular filtration of C(A) 
by Lemma 14.171 Note that by Theorem 14.71 we have Q'^.Qr = Qr-'u if R 
is a rectangular partition and partition u C R. Thus we have 

(20) Qx = Qm{x)~'€(x) = Qe:{x)-Q'n{x) 

= (---((/;./,_i)*./,_2)*.-- - ./2)*./i. 

□ 
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Example 4.22. Let us look at the A in Example 14.151 By this theorem, 
((QiQ(23))*(5(54))*(5(65) is equal to Q\ up to a non-zero scalar multiplication. 
We can use our vertex operator formula to find the Qi^k")- 

Remark 4.23. 1. The coefficient c^(a) can be explicitly evaluated just 
as we have done in the proof of Theorem 14.71 

2. The argument of Theorem 14.211 also works for Macdonald symmetric 
functions, thus we would have a vertex operator realization for Macdonald 
functions of general shapes if we had that for rectangular shapes. 

As an application, we have a combinatorial formula for Jack functions 
of general shapes. 

Corollary 4.24. For a partition X, let Rx = (i?i,i?2,--- , Rs) be the 
rectangular filtration of X, denote fi = Qr^, then we have 

where c^(a) is a non-vanishing rational function of a, gji.^^i{a) is defined by 
Definition \3.3[. and the sum runs over sequences of partitions ^ = {(^ ,£} , ■ ■ ■ 
associated with i = A'') such that f = c' C for 

i = s-l,--- ,2,1,0 with t = {) andi^ = i^. 

Proof. Note that Corollary 13.41 implies the expression for Q\ with the 
help of (I19p . In the computation we introduced the variables ^*'s to simplify 
the expression and we iteratively use the following identity 

which can be checked by the computation: {p^''^)* -p^^ = (ia)™' wij! 

for ruj < m. □ 
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